Skip to content

fix: check trimmed text for heading marker so note bodies reach embeddings - #14

Open
arimu1 wants to merge 1 commit into
obra:mainfrom
arimu1:fix/6-embed-note-body
Open

fix: check trimmed text for heading marker so note bodies reach embeddings#14
arimu1 wants to merge 1 commit into
obra:mainfrom
arimu1:fix/6-embed-note-body

Conversation

@arimu1

@arimu1 arimu1 commented Jul 19, 2026

Copy link
Copy Markdown

Problem

Semantic search (kg_search) effectively ignores note bodies. Excerpts show the # Title heading instead of body text, and any note whose title doesn't literally contain the query keyword is unreachable via semantic search — even when the body is a strong match.

Root cause

gray-matter leaves a leading newline after stripping the YAML frontmatter, so content.split(/\n\n+/)[0] is typically '\n# Title', not '# Title'. Two call sites checked p.startsWith('#') before trimming, so that leading-newline heading was never recognized as a heading:

  • src/lib/store.ts:283 (firstParagraph, used for searchVector excerpts) — the heading slips through as the "first paragraph" shown in results.
  • src/lib/embedder.ts:37 (buildEmbeddingText) — higher impact: split[0] was taken unconditionally with no heading check at all, so the embedding text ends up as title + tags + "# title". The note body never enters the embedding vector, which collapses semantic recall to title matches only.

Verified this against the current code on main (commit 1d2481e) before making any changes.

Fix

Extracted a shared firstBodyParagraph() helper (src/lib/text.ts) that trims each candidate paragraph before checking for a leading #, per the fix suggested in the issue. Used it at both call sites:

  • store.ts's firstParagraph() now delegates to the helper before length-capping.
  • embedder.ts's buildEmbeddingText() now uses the helper (and trims the result) instead of an unconditional split[0].

Testing

  • Added test/text.test.ts: direct unit tests for firstBodyParagraph, including the gray-matter leading-newline case.
  • Added a regression test in test/embedder.test.ts asserting buildEmbeddingText includes the body paragraph (not the # Title heading) when content has a leading newline before the heading.
  • Added a regression test in test/store.test.ts asserting searchVector excerpts contain the body paragraph, not the heading, for the same leading-newline shape.
  • Ran the full suite: npx vitest run → 13 files, 92 tests, all passing.
  • Ran npm run build before and after the change and diffed the output: identical 32 pre-existing tsc errors in graph.ts/mcp/index.ts/embedder.ts (unrelated dependency-version type issues already present on main), confirming this change introduces no new type errors.

As noted in the issue, existing indexes will need kg index --force after upgrading, since embedding content changes.

Fixes #6.


This PR was developed with AI assistance (Claude Code, model Claude Fable 5) with human review.

…dings

gray-matter leaves a leading newline after stripping YAML frontmatter,
so a naive split('\n\n') on note content yields '\n# Title' as the
first paragraph. Both firstParagraph() in store.ts (search excerpts)
and buildEmbeddingText() in embedder.ts checked p.startsWith('#')
without trimming first, so that leading-newline heading was never
recognized as a heading — it was treated as the "first paragraph" and
used verbatim. In embedder.ts this meant note bodies never entered the
embedding vector at all (embedding text was title + tags + title),
collapsing semantic search recall to title matches only.

Extracted a shared firstBodyParagraph() helper (src/lib/text.ts) that
trims each candidate paragraph before checking for '#', and used it at
both call sites.

Added unit tests for the helper directly, plus regression tests for
store.searchVector() excerpts and Embedder.buildEmbeddingText()
against gray-matter-style leading-newline content.

Fixes obra#6.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Semantic search ignores note body — firstParagraph misreads gray-matter output, embeddings built from title+tags+title

1 participant